@Septa2112

If these changes are acceptable to the community, I'd like to modify other places that depend on logits_all == true, such as

llama.cpp/examples/imatrix/imatrix.cpp

Lines 516 to 520 in 90db814

	// TODO: use batch.logits to save computations instead of relying on logits_all == true
	if (llama_decode(ctx, llama_batch_get_one(tokens.data() + batch_start, batch_size, j * n_batch, 0))) {
	fprintf(stderr, "%s : failed to eval\n", __func__);
	return false;
	}

Hi, I wrote that comment in #6122, but after studying imatrix.cpp more I think for imatrix specifically that always using all logits is fine, because otherwise the hidden state going through the FFN of the last layer would get trimmed, which would not be good for the importance matrix calculation.

Though it's true that it could avoid using logits_all, while still specifying that all logits should be output, this is already what is internally done when using logits_all (still, removing it could be worthwhile).

For the strided perplexity calculation (aka perplexity_v2) which is modified here, the main reason this was not updated to use an allocated llama_batch is because I was not sure which output should be kept. For the perplexity() function, it's simply the second half of the logits, while for this one, I'm not sure.

I see in b0c6ad7 that you've simply made it ask for all outputs, same as using logits_all. This is a good start because using an allocated llama_batch is necessary for more control over which outputs to keep.

If you do figure out which outputs to keep, know that the logits in the buffer returned by llama_get_logits are all contiguous (no holes). This means if outputs are excluded, then the offsets in that buffer will need to be corrected. llama_get_logits_ith abstracts this away and uses the original indices used in the llama_batch, but can only be expected to return a pointer to the logits of a single token at a time (although technically, this is all the same buffer, so it's also possible to use it simply to get the offset of a contiguous range).

Hopefully this helps!

@compilade Thanks a lot for your suggestion! I will carefully consider your advice and propose changes related to imatrix.cpp in another separate PR.

In the current PR, I only plan to modify the perplexity_v2 section. The remaining two parts, I still need time to consider how to modify them. :)